I Never Forget a Face

Both the Oxford Dictionary of Quotations (7th edition, 2009) and the Oxford Dictionary of Modern Quotations (first paperback edition, 1992) give Leo Rosten's People I have Loved, Known or Admired as the source of this quote.

Leo Rosten was a Russian–born American humorist. He wrote several film scripts, but is best remembered (according to Wikipedia) 'for his stories about the night–school "prodigy" Hyman Kaplan, written under the pseudonym Leonard Q. Ross ... published in The New Yorker from 1935 and collected in two volumes published in 1937 and 1959.' Rosten died in 1997, aged 88.

I was curious to know who had been the object of Groucho's wit, and this meant I had to get hold of a copy of People I have Loved, Known or Admired. (You may consider this to be somewhat obsessive – and you may well be right.) It wasn't hard to find a used copy on the Internet, and it wasn't expensive. A fairly substantial volume, published in 1970, it gives pen–portraits of thirty–odd people, of whom Groucho and Winston Churchill are the most famous. Rosten clearly knew Groucho, but he doesn't claim to have been present when the remark in question was made. He tells the story thus:

'When a tipsy delegate to some convention of cretins slapped Marx on the back with that hallowed American anthem, "You old son of a gun, you probably don't remember me!" Marx impaled the oaf on his misanthropic glare and replied, "I never forget a face, but in your case I'll be glad to make an exception."'

So after all that, we don't really know who Groucho said it to. It's not exactly the most convincing attribution I've ever seen ... but if it's good enough for Oxford Dictionaries, it's good enough for me.